A hierarchical Bayesian model for improving wisdom of the crowd aggregation of quantities with large between-informant variability

The wisdom of the crowd technique has been shown to be very effective in producing judgments more accurate than those of individuals. However, its performance in situations in which the intended estimates would involve responses of greatly differing magnitudes is less well understood. We first carried out an experiment to elicit people’s estimates in one such domain, populations of U.S. metropolitan areas. Results indicated that there were indeed vast between-subjects differences in magnitudes of responses. We then proposed a hierarchical Bayesian model that incorporates different respondents’ biases in terms of the overall magnitudes of their answers and the amount of individual uncertainties. We implemented three variations of this model with different ways of instantiating the individual differences in overall magnitude. Estimates produced by the variation that accounts for the stochasticities in response magnitude outperformed those based on standard wisdom of the crowd aggregation methods and other variations.